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1. Introduction 

Let M" be an n dimensional manifold isometrically immersed into the space form 
iV"+™(c) of constant sectional curvature c. Define the normalized scalar curvature 
p and p-^ for the tangent bundle and the normal bundle as follows: 

2 " 

^ ' l=i<3 

(1) 



^ ' \ l=i<j l=r<s 



Date: April 9, 2006. 

The second author is partially supported by NSF Career award DMS-0347033 and the Alfred 
P. Sloan Research Fellowship. 



2 



TIMOTHY CHOI AND ZHIQIN LU 



where {ei,-- - ,€„,} (resp. {^i,-- - ,£,m}) is an orthonormal basis of the tangent 
(resp. normal space) at the point x £ M, and R, R-^ are the curvature tensors for 
the tangent and normal bundles, respectively. 

In the study of submanifold theory, De Smet, Dillen, Verstraelen, and Vrancken [5] 
made the following DDVV Conjecture: 

Conjecture 1. Let h be the second fundamental form, and let H = — trace h be 
the mean curvature tensor. Then 

p + p^< \H\^+c. 

A weaker version of the above conjecture, 

P<\H\^+c, 

was proved in "2" . An alternate proof is in [llj . 
In [S] , the authors proved the following 
Theorem 1. Ifm = 2, then the conjecture is true. 

In this paper, we prove the conjecture in the case n = 3, which is the first non-trivial 
case. The proof is quite technical and, like [71[5j, some non-trivial linear algebra is 
involved. We also point out a relationship between the DDVV conjecture and the 
comass problem in Calibrated Geometry. We believe that the method used here 
will be useful in Calibrated geometry. The more general cases of the conjecture will 
be treated in our second paper [10] . 

Let X € M he a, fixed point and let (h'^j) («, J = 1, • • • , and r = 1, • • • , to) be the 
coefficients of the second fundamental form under some orthonormal basis. Then 
by Suceava [12j . or [6], Conjecture[T]can be formulated as an inequality with respect 
to the coefficients as follows: 



E E (M.-h^,r+2nY: E 



m n m n 

r—l l—i<cj r—1 l—i<cj 

(2) 

m n 



2\ 2 



E E (T.'^'^'^^'^-Mk^k) 

l=r<s l=i<j \k=l 



Suppose that Ai, A2, • ■ • , Am are n x n symmetric real matrices. Let 

n 

E4' 

where (oij) are the entries of A, and let 

[A, B]^ AB - BA 

be the commutator. Then the equation in terms of matrices, can be formulated 
as follows 
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Conjecture 2. For n,m> 2, we have 

m 

(3) (^iiA.in2>2(^ii[A,,^]in. 

r— 1 r<s 

Fixing n,m, we call the above inequality Conjecture P(n,m). 

Remark 1. For derivation of ([2]), see 6, Theorem 2]. Note that the prototype of 
the matrices are the traceless part of the second fundamental forms. 

The main result of this paper is: 

Theorem 2. Let Ai (i — 1, ■ ■ ■ , m) &e 3 x 3 symmetric matrices. Then we have 

m 

(4) (J^IIA.II^f >2^(||[A„^,]||2). 

i—l i<j 

That is, the Conjecture P(3, m) for to > 2 is true. 

In the second part of of the paper, we discussed the cases when the equahty is vaHd. 
In particular, we classified all minimal 3-folds such that the equality of Conjecture[T] 
is valid at any point. Such kind of 3-folds are a special kind of austere submanifolds 
defined in ^ . Austere 3-folds were locally classified in [H |3] , when the ambient 
space is the Euclidean space or the unit sphere. We provide a similar classification 
in the case of ambient space being hyperbolic, using the similar method of theirs. 
Restricting to the situations in this paper, we also give a more straightforward way 
to classify austere 3-folds. 

Acknowledgment. We thank Suceava for his bringing this very interesting ques- 
tion, as well as many references, into our attention. We also thank him for many 
useful comments, which were absorbed into the current version of this paper. Deep 
thanks to R. Bryant, who introduced the paper [3] and made many useful comments 
to me; and to C. Terng, who explained to me, in great details, of the submanifold 
geometry in Minkowski spaces; and to M. Dajczer and F. Dillen, who pointed out 
to the second author, that when equality is valid the 3-fold needs not to be minimal. 
Without their helps, the paper won't be in its current form. 

2. CoMASs IN Calibrated Geometry 

Before making further analysis of Conjecture [U we introduce the concept of the 
comass of a p form in Calibrated Geometry (cf. [3]). 

Consider Euclidean space R" with orthogonal basis ei, • • • , e„ and dual basis dxi — 
e*. Let I — (ii, • • • ,ip) denote a multi-index with ii < ■ ■ ■ < ip. Let 

f^Yl ^ ■ • • ^ Sip 

be a p-covector (constant-coefficient p-form). The comass \ \(p\\* of is given by 
\\ip\\* — max{(/j(^) I ^ is a p-plane}. 
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For a differential form on a Riemannian manifold M, its comass ||v||* is given by 

ii<^ir = sup{ii<^,irixGM}. 

X 

In [7], Gluck, Mackenzie, and Morgan initiated the study of the comass of the first 
Pontryagin form on Grassmann manifolds. Later Gu [5] generalized the results, 
and Harvey [3] gave a simplified and unified proof. 

Their results are listed as follows: 

Theorem. The comass of the first Pontryagin form ip on the Grassmann manifold 
G{n,m) is as follows: 

(1) \\ip\\* IS ^/3/2 for n = 3,m = 6, 4/3 for n ^ 3,m > 7, and 3/2 for 
n = 4,TO > 8 7 J 

(2) IS 3/2 for n> 4 or 771 > 8 [8]. 

The definition of the comass, in the context of the comass of the first Pontryagin 
form, can be formulated as following problem: 

Let A,B he two m x n matrices. Define 

{AB} = AB^ - BA^. 

Let 

ip{Ai A ^2 A ^3 A A4) 

- -^tr ({AiA2}{A3A4} + {AM{A2A4} + {A2A3}{AM) 

for m X n matrices v4i,v42,^3, and A4. The comass of is defined to be the 
maximum of the right-hand side of the above under the condition that Ai^ A2, A^, 
and are orthonormal. 

Conjecture is similar to the above comass problem in that both problems are 
related to the commutator of matrices. In fact, P{n, 3) can be reformulated as 
follows: let A^B^C he nx n symmetric matrices such that 

\\Af + \\B\\^ + \\C\\^^l. 

Then 

\\[A,BW + \\[B,C]\\^ + \\[C,AW <\. 

The major difference between these two problems is that they have different invari- 
ant groups. The comass problem is invariant under 0{n) x 0{Am) (see [7] or [5] for 
details). On the other hand the invariant group for the DDVV Conjecture is not 
known. In the next section, we proved that the invariant group contains the group 
0{n) X 0{m), which allows us to solve the conjecture for n — 3. 

Although our method is quite different from [7j or [8] , we got hints from the proof of 
both papers. In particular, we learnt that the larger the invariant group, the more 
reductions we can do on the matrices. Thus the authors strongly believe that the 
results of both problems should be parallel after more information of the invariant 
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group of the DDVV Conjecture is found. It would also be very interesting to study 
the submanifolds "calibrated" when the equality of the conjecture is valid at any 
point. 

3. Invariance 

Let Ai, - ■ ■ , Am he nx n symmetric matrices. Let G — 0{n) x 0{m). Then G acts 
on matrices (Ai, • • • , Am) in the following natural way: let (p, q) G G, where p, q 
are n x n and m x m orthogonal matrices, respectively. Let q = {qij}- Then 

(p,0) • (^i,--- ,Am) = {pAip-\--- ,pAmP~^), 

and 

m m 

(0,(7) • (^1, • • • ,Am) = C^qijAj, ■ ■ ■ ,'^qmjAj). 
It is easy to verify the following 

Proposition 1. Conjecture P{n,'m) is G invariant. That is, in order to prove 
inequality ^ for {Ai, ■ ■ ■ ,Am), we just need to prove the inequality for any 7 • 
{Ai, • • • , Am) where 7 € G. 

□ 

As a consequence of the above proposition, we have the following interesting 

Theorem 3. Let n > 2 be an integer. If P{n, ^n{n — 1) + 1) is true, then P{n, m) 
is true for any m. 

Proof. Obviously, if P{n, 1) + 1) is true, then P{n, m) for any m < ^n{n—l) 

is true because we can set the excessive A^'s to be zero. To prove the case when 
m > in(n — 1) + 1, we use the math induction. By Proposition [Tl we can assume 
that at least one of the matrix, say Ai, is diagonalized. Because the off-diagonal 
parts of the matrices form a ^n{n~ l)-dimensional vector space, we can find a unit 
vector (a2, • ■ ■ , am) such that 

a2A2 + h amA 

is diagonalized. Extending the vectors (1, 0, • • • ,0) and {0,a2, - ■ ■ , Om) to an m x m 
orthogonal matrix q, we can assume, without loss of generality, that A2 is diagno- 
lized. In particular, [^1,^2] — 0. 

Replacing Ai , A2 by cos aAi+ sin a A2, — sin aAi+ cos a A2 for suitable a, respec- 
tively, we can assume that Ai J- A2. 

Assuming that P{n, m — 1) is true, we have 

(6) {\\A,+A2\\' + Y.\\A\\Y>2Y.\\[A^+A2,A,W + 2 ^ m,A,W, 

i—3 i—3 3<i<j 
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and 

m m 

(7) + >2^||[Ai-A2,A.]||' + 2 J2 

i—3 i— 3 3<i<j 



1 12 



Since IIA1+A2IP = ||^i-^2|P = + IIA2IP, adding the above two equations, 

we have 



i— 1 \ i— 3 3<'i<j 



1 12 



Since [^1,^2] = 0, the theorem is proved. 

□ 

Corollary 1. //P(3,4) is true, then P{3,m) is true for m > 2. Moreover, Theo- 
rem\^foUows from P(3,4). 

□ 

4. Proof of P(3,3) 

We first give a proof of the result of De Smet, Dillen, Verstraelen, and Vrancken [5]. 
By our setting in the previous section, the result can be formulated as 

Proposition 2. Let A, B be n x n symmetric matrices. Then we have 

i\\A\\' + \\B\\'f>2\\[A,BW- 

Proof. By Proposition [l] we may assume that one of the matrices, say A, is 
diagnolized. Let 

^bii ■■■ bir, 









-1 













with bij = bji. Then we have 



bni ■ ■ ■ br, 
\\[A,BW^2Y,{X,-\,fbl 



Obviously, we have 

n 

(8) {K-Xjf < 2^2^1 = 2 

k=l 

Thus we have 

\\[A,BW<A\\A\\'T.bl<m\\'-\\B\ 

i<j 

and the proposition follows from the above inequality. 



□ 
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Before proving P(3, 3), we establish the following 

Lemma 1. Let pi,p2,P3 > be three real numbers. Let Ai,A2,A3 be three other 
real numbers such that Ai + A2 + A3 = and A| + A| + A3 = 1. Then we have 

3 ^ 

(9) X! ^kPk > g (Pl + P2 + P3 - V {Pl +P2+ P3,f - 3(piP2 + P2P3 + PzPl)) , 
k=l 

and 

3 2 

(10) ^A^pfc < -(pl+p2+P3)• 
fc=l 

Furthermore, we have 

^ (Ai - \jfpk 

(11) i<3;kyii,j 



<Pl +P2+P3 + V (Pl +P2 +P3)^ - 3(j3iP2 +P2P3 +P3Pl)- 

Proof. Under the restrictions on Ai,A2,A3, the maximum and minimum of the 

function 

3 

/(Ai,A2,A3) = ^A2.pfc 
fc=i 

exist. Let's use the Lagrange multiplier method to find the maximum and minimum 
values. Consider the function 

3 33 
F = ^ Xlpk ~ Mi(^ A,) - M2(E - 1). 

k=l k=l k=l 

where Mi,M2 are multipliers. At the critical points, we have 

(12) 2AfePfe - Ml - 2M2Afe = 

for fc = 1, 2, 3. Multiplying by Afe in (fT^ and summing up, we have 

3 

'^Xlpk = M2, 

fe=i 

at the critical points. Thus the maximum and minimum values of / are the values 
of ^l2■ 

We claim that //2 satisfies the equation 

(13) (m2 -Pi)(M2 -P2) + (M2 "P2)(M2 -P3) + (M2 -P3)(M2 -Pl) = 0. 

To see this, we first assume that /i2 is one of pk (fc — 1, 2, 3). Then by ([H]), Mi — 0- 
Since at least two of the three Afc's are not zero, there is / ^ fc such that M2 = Pi- 
Thus pS]) is satisfied. On the other hand, if M2 Pi,P2,P3, then from (|12p . we 
have 

2Afe = ^^ 
M2 -Pk 

for fc — 1,2,3. Thus M2 satisfies (fT3|) . 
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By solving /i2 wc get the maximum and minimum values of the function /, which 
proves dHI) and (fTO|l . Since 

(A.-A,)2=2~3AL 
we get and the lemma is proved. 

□ 

Proof of P(3,3). We let the entries of A,B,C be aij,bij and Cij, respectively. 
Using Proposition (TJ we can assume that A is diagonahzed. That is, = if 

We let = all + ^22 + ^33- Then since m = 3, inequality (jl]) can be written as 



(14) {f + \\B\\' + \\c\\r > E ('^^ " + 4) + mB,cw. 

i<j, 

Let 

(15) / = vUbh + 4) + Vi{bl3 + 4) + Vl{bl2 + c?2)- 
Then ((T4l) can be written as 

(16) t^ + t2(-2(||i3||2 + ||qn + 4|H|2 + l2/) + (||i?||2+||C|n2-2||[i?,C]|p>0, 
where /it = (&11, 622, ^33, cn, C22, 033)^. 

Let 

&i2 = f3C0sa3, C12 = r3 sinQ!3, 

(17) &13 = 7-2 cosa2, ci3 = r2 sinQ!2, 

&23 = ^"1 cosai, C23 = sin ai , 

Using ([9|) of Lemma [U the above inequality is equivalent to 

(18) t'' + (2|Hp - 4V^) + (||B|p + \\C\rf - 2||[i?, C]|p > 
for any t, where 

/ 2 I 2 I 2\2 0/ 2 2 I 2 2 I 2 2\ 

If 2||/i|p - > 0, then inequality (dH) follows from Proposition [2l If 2||//|p - 

4^7710 < 0, by choosing suitable t minimizing the left hand side of (llSp . we get 

(19) (||S|P + \\C\rr - 2||[i?, > (2V^- ll/illY, 
whenever 2^mo ~ \ > 0. 

Let ^ e R'^ and let P : — > R'^ be a linear map. We define 

(^'13C23 - 013623^ 
&12C23 - C12623 
&12C13 - C12613/ 
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and 



Then we have 




-Cl2 





-bl2 


bi2 








-Cl3 


-bi3 





6l3 


C23 


-C23 





-^23 


^23 



||[S,q|P = 2||PM + ^lP 



Using the above terminology and expanding (|19p . we get 

(20) I p(r? +ri + rl + V^-\ \Px\ \^) ~ 2M l^Px + 3ao - | ICI P > 0, 

where x — | and ctq — ?'i^2 + ^2*^1 + ^i'^i ■ 

Since (0 and (HO]) are equivalent, ^ is valid if = 2^m^^. 

If 



or if 
but 



\\Px\r < 0, 



/mi- \\Px\\^ > 0, 



2,2,2 



,2 -r -3 -r Or C^Px < 0, 

then the minimum of the left hand side of (|20p is achieved at either ^ — 0, or 
— 2^mo. In view of ([TO]) and the equation below, in either case, the left 
hand side of (|20p is nonnegative. Thus (|20p is valid in the above two cases. Finally, 
if 

rl + rl + rl + ^/TE^ - ||Pa;|p > 0, 

and 

(21) < e^Pa; < %/2 • ^{rj + + rl + - | \Px\ P), 

then JSOl) is valid if 

i^Px)^ < {rl +ri+rl + ^-\ \Px\ f){3<Jo - 1 1^1 P). 



But by ([2T|) . it suffices to prove 

Thus P(3, 3) follows from the following 
Lemma 2. Using the above notations, we have 

(22) ^o-iieii'>o, 

and 

(23) \\P^^\\V2 - ^<2ao. 

Proof. Under the polar coordinates, we have 

(rir2 sin(ai - 
rir3s'm{ai ~ Q3) 
r2r3 sin(a2 - 03)^ 
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Thus (I22| follows. To prove ([23]), we first write 



(2(ci2 + &i2) C12C13 + 612613 -C12C23 - 612623 
C12C13 + 612613 2(0^3 + 6^3) C13C23 + 613623 
-C12C23 - 612623 C13C23 + 613623 2(C23 + ^23) 

In terms of the polar coordinates, we have 

(2r| r2r3 cos(a2 - q;3) -rir3 cos(a3 - ai)^ 

r2r3 cos(q;2 - as) 2r| rir2 cos(q;i - (32) 

-rir3 cos(a3 - ai) rir2 cos(q;i - 02) 2ri 

from which we have 

(24) llP^eil' = Sr^r^rl (sin2(ai - (12) + sin2(a2 - 03) + sin2(a3 - ai)). 
Using Lagrange multiplier method, we get 

sin^(Q!i — Q!2) + sin^(Q;2 — 0:3) + sin^(a3 — ai) < 9/4. 

Thus we have 

||P^eil<^™. 
From the above inequality and (f23|) . it suffices to prove that 

(25) ^rir2r3-\/2-^<2ao. 

Let a, 6, c be positive numbers. Then by expanding the expression, we have 

{a + h + cf> 6{a^b^ + b^c^ + c^a^) + 4(a^6 + 06^ + b^c + bc^ + c^a + ca^ 
By using a^b + ab^ > 2a^b^, etc, we get 

{a + b + c)'^> Uia^b"^ + b^c^ + c^a^). 

Using the above inequality, we have 

(26) at > Urt^rtirt + 4 + r^) > Mr^r^r^o- 
Thus 

2(To > 2^/TArir2rz^fm^ > — ^r-ir2r3 • \/2 • ^m^, 
and (l25l) is proved. 
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5. Proof of P(3,4). 



Now we begin to prove P(3,4). That is, we want to prove that, for traceless 
symmetric 3x3 matrices A, B,C, D, we have 



As before, aij,bij,Cij,dij represent the («,j)-th entries of the matrices A,B,C,D, 
respectively. Before proving the inequaUty, we have the fohowing result which gives 
some reduction of the matrices: 

Lemma 3. Without loss of generality, we may assume 
(1) A is diagonalized; 



Proof. By Proposition [U we may assume that A is diagonalized. Since the off- 
diagonal parts of a 3 X 3 matrix form a three dimensional space, we can find real 
numbers ai, a2, and with a\+ a\-\- = 1 such that the entries (1, 3) and (1, 2) 
of the matrix aiB + + a^D are zero (after a possible permutation). We now 
extend (ai, 0:2, as) to a 3 x 3 matrix 



Replacing B, C, D by jiB + 72C + 73!?, PiB + P2C + /^sZ?, aiB + aaC + asD, 
respectively, we get di3 = di2 = 0. Finally, by choosing suitable a and replacing 
B, C with cos a B + sin a C and sin a B — cos a C respectively, we may assume that 

Cl2 = 0. 



The proof of P(3, 4) will be similar to that of P(3, 3). As in the proof of P(3, 3), if 
we let t = \ \A\\ and let A' = A/t, then we have 



(27) 



{\\A\\' + \\B\\^ + \\C\\' + \\D\\'f 
>2i\\[A,BW + \\[A,CW + \\[A,DW 

+ \\[B,c]\\' + \\[B,D]\\' + mD]\n. 



(2) di3 = di2 = 0; 

(3) C12 = 0. 




□ 



(28) 



{t' + \\B\\' + \\C\\' + \\D\\'r 
>2t\\\[A',B]\\^ + \\[A',CW + \\[A',D]\\^) 
+ 2{\\[B,CW + \\[B,D]\\^ + \\[C,DW). 



According to Lemma |3l we assume that A is diagonalized, and we have 



B = 



ffil h b2\ /fi4 C2\ /fl7 \ 

lbs (12 61 , C = ci , \0 /is di 

\&2 bi (12,) \C2 Ci (Iq) \0 di flgj 
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Let 

Pi = ^Jbl + cl+dl 

P2 = \Jbl + cl, 
(29) V3 = \b3.\, 

2 2 I 2 2, 22 
di =PiP2 +P2P3 

™i = (Pi +-P2 +p'iT ^ 3cti, and 
= • ■ ■ ,^9)^- 

By Lemma [U we have 

\\[A\B]\\^ + \\[A',C]\? + \\[A\D]\\^ < 2{pI+pI+pI + V^). 

If - ll^lp < 0, then ^ is trivially true. Otherwise, like IH]), ^ is 

equivalent to the following 



^3Q^ Vdl^lP + \\C\\' + \\D\\^r - 2||[S,C]|P - 2||[i3,i?]|P - 2||[C,Z?]|P 










-63 


b3 



















-62 





62 












-ci 







-61 


61 





0^ 






















bs 




















-b2 





^^2 , 


dl 



















bj 

















































C2 , 











di 


-di 









cj 




Assume that 

||[i3,C]|p = 2||F3M + 6llM|[S,i?]|P = 2||P2M + 6llMl[C,Z?]|P = 2||Pi^ + Ciir 
Then we can write out the matrices explicitly as follows: 
'0 

P3 = I C2 

Cl 



P2 



Pi = -C2 C2 , Cl = 



A straightforward computation gives 

^IPs = (fe3ClC2, 63C1C2, -263C1C2, -26263C1 + 6163C2, 

6263C1 - 2616302, 6263C1 + 6163C2, 0, 0, 0), 
e2''P2 = (0,0,0,0,0,0,-26263^1,6263^1,^3^1), 





(31) 



ON THE DDVV CONJECTURE AND THE COMASS IN CALIBRATED GEOMETRY (I) 13 



We have 

W^IPs + ^lP2 + ^IPi\? = (^hl{c\cl + hld\ + hlcl + hlcl - 6162C2C2). 
Therefore we have 

WilP^i + ilP2 + i[Pi\? < 7(6? +cl+ dl){bl + cl)bl 
from which we have 

(32) \\^IP3+ iIP2 +^IPl\\< V7PIP2P3 ■ 

Usmg the definition of pi,p2,P3,ai, mi, and ^i, (f30|) is equivalent to 

3 

i2{pl+pl+pl)+M\y-4Y^m^iH^\\' > i2y^^~\\fi'\\f if 2V^-||Ai2|| ^ 0^ 

Extending the above expression, we get the following quadratic inequality: 

33 3 

(33) (p?+P^+P^ + mi-^||P,x|n|H|2-(^erP.)^IHI+3ai-^||e.|P>0. 

i—1 i—1 i—1 

Again, using the same method as in Lemma [21 we can prove that 

(34) Y.\M'<a,. 

i=l 

From (1201) and we know that ^ is true if | | = or a/2 • If the 

minimum value of the quadratic is reached by one of the two points, then the 
inequality is proved. If the minimum of the above is reached by some point between 
(0, V2 • ^mi), then we must have 

(1) pl+pl+pl+m,~ I \P,x\ |2 > 0; 

(2) < Ell lli^.^ll < V2 ■ ^M+pI+pI + - Ell m^W')- 

Like in the proof of P(3, 3), it suffices to prove that 

3 

^iiefp.ii •%/2- ^< 2ai. 

1=1 

By (1311), it suffices to prove that 

(35) V7piP2P3 ■ V2 ■ < 2cti . 

By (im, the above inequality is true, and thus P(3, 4) is proved. 

□ 

Proof of Theorem l2j Since P(3,4) is true, by Corollary ITJ P(3,m) is true. 
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6. The equality cases 



□ 



We first establish the following 

Proposition 3. Let A, B be n x n symmetric matrices. If 

{\\A\\' + \\B\\'r = 2\\[A,B]\\\ 
then there is an orthogonal matrix Q and a real number A such that 



A = QA!Q'^, B = QB'Q', 



ir\T 



whe 



A' = 



/A 



V 



-A 



B' = 



0/ 



/ ±A 
±A 



V 



0/ 



Proof. Without loss of generality, we assume A is diagnolized, then we are in the 
same situation as in Proposition [2l In order for the equality of ([8]) to be true, we 
assume that 



(Ai-A2)2 = ^A 



k=l 



Let A = Ai. Then we must have A2 = —A, A^ = for k > 2, and bij 
{i,j} — {1,2}. Finally, a straightforward computation gives 612 — ±A. 



unless 



□ 



The following theorem shows that the equality cases of P(3, m) is quite restrictive. 
Theorem 4. Let A^ (i = 1, • • • , m) fee 3 x 3 traceless matraces. If 

m 

(36) iJ2\\A,\ff^2{J2\\[A,,AM'), 

i—l i<j 

then up to an element 7 S G, we have Ai — for i > 2 

^A \ / ±A 

Ai = I -A 1 , and A2 = l±X 



Proof. By Propositon [1] up to an element of G, we have Ai = for i > 4. When 
the equality holds, the equality in pO|) is reached: 



(37) 



Vi\\B\\^ + ||C|P + ||D||2)2 - 2\\[B,CW mB,DW - 2||[C,i?]|| = 



2Wmi 



HP 



We claim that one of the matrices A,B,C,D must be zero (up to a G action). 
In fact, if 2y/rni — — 0, then A = 0. So the claim is valid. If fi — 0, then 
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from (l33|) and (p4|) . tri = 0. So two of pi,p2,P3 are zero. Combining with /i = 0, 
we see that (1). if pi — 0, then D = 0; (2). if P2 = Ps = 0, then B,C,D are 
proportional. Therefore after a G action, one of the matrices B, C, D must be zero. 
FinaUy, if < H/xlP < 2y/rrii, then from ((35)) . again cti = 0, and we are in the same 
situation as in the fi = case. 

Since one of the four matrices A,B,C,D must be zero, we get the equahty case 
of (HH): 

(38) (iisip + iicin^ - 2\\[B, c]\\' = (27^- M\^r. 

We essentially repeat the previous proof by claiming one of the two matrices B, C 
must be zero. In fact, if 2^mo — = 0, then A = 0. If < | p < 2^mo, then 
(To defined in (|23p must be zero. Therefore two of the ri,r2,r3 must be zero and 
B, C must be proportional. Thus after a G action, one of the matrices B, C must 
be zero. 

In summary, from the above argument, we know that after a G action, only two 
of the m matrices Ai, • • • , may be non-zero. Using Proposition [31 we get the 
conclusion of the theorem. 

□ 

Now we characterize submanifolds of space form N^+"^ (c) such that p + = 
|ff P + c at every points. By Remark[T]and Theorem^ we have the following 

Theorem 5. Let be a submanifold of the space form A^'^+™(c) such that 

(39) p + p^^\H\^+c. 

Let the local orthogonal frames ofTM he 60,61,62 and the local orthogonal frames 
of T^M 6e ^Oi ■ ■ ■ 5 Cm-i • the matrices of the second fundamental form corre- 
sponding to be Ai for Q < i < m — \. Then Ai — if i > 2, and 

/Ao \ . /^i+Ai \ ^ /A2 ±p 

io = Ao , ii = Xi- p \ , A2= \±p X2 

V Ao/ V ^1/ V ^2 
where Ao, Ai, A2 and p are local functions on M . 

□ 

One can easily find a large classes of 3-folds satisfying Theorem [5] One important 
class of such 3-folds were given in [6], including the totally umbilical submanifolds. 

On the other hand, even the classification of minimal 3-folds satisfying (|39p will 
be very interesting and fruitful. So for the rest of the paper, we will only discuss 
the cases when M is minimal. The classification is related to the so-called austere 
submanifolds. At the end of this section, we will give the definition. If M is an 
austere submanifold, then Ai = 0, < i < 2 in Theorem [SJ 

Let 60, ■ • • , 62+r be orthonormal frame fields and let woi ■ ' ' 1 be orthonormal 
coframe fields of the space form N'^^^{c). The Cartan structure equations are as 
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follows 



(40) 
(41) 



dujj — —uJiK A lok', 

dojjj = —LjjK A ujkj + cujj A ujj, 



where the upper case Roraan letters I, J, K, - ■ ■ range from through 2 + r, and 
t^/j = -t^j/- 

We only need to consider the cases c — 0, 1,-1 after rescaling. We use the fol- 
lowing standard notations: let iV3+'^(0) = N^+'-{l) = 3^+'' C R^+'', and 
N^+^{~1) = C R^+'''\ where R^+'''^ is the Minkowski space endowed with 
the following metric 

(42) dxl + • • • + dxl_^_J. — dxl^^, 

and H^^^ is defined as the hypersurface xf + ■ ■ ■ + — x^+r = — 1 of R'^+'''^. 

Let M be an embedded austere 3-fold in N'^^^{c) (where c = 0, 1,-1) such that 
eo, ei, 62 are the tangent vector fields and 63, • • • , e2+r are normal fields of M. In 
what follows, we will use lower case Roman letters for the "normal" index range 
S < a,b, c < 2 + r, except for k, where they range from through 2. 

Along the submanifold M, the Cartan structure equations (j40ll . (|4T|) can be re- 
written as 

(43) duji — —ujij A uij; 

(44) = -ujaj Aojj; 

(45) dujij = —ujiK A LUKj + cLUi A ojj] 

(46) dhJaj = -i^aK A UJKj- 

By the minimality and Theorem [5l we may assume that 



(47) 



= 0, 3<a<2-hr. 



From (|33]) and (gT]), we have 
(48) 



LOal ALl>i+ LOa2 A CJ2 = 0. 



Thus LOaX 



and LOal are linear combination of uj\ and 102 ■ Let 




and let 




Then by the minimality of M, we 

(49) TTa = ZqW, 

for some complex function Zq. From (|46p . we have 







t^al A CJio - CJq2 a W20- 



^The equations II49I I and USUI are essentially from [T] §4]. 
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If for some a, Za ^ 0, then a-'io,'^20 are the linear combinations of L0aiTL0a2 and 
hence (^1,^2- In particular, we have 

(50) a;io(eo) cj2o(eo) = 0. 
We let 

and assume that 

TTq — ZqUJ + huj 

for complex functions zo and h. Since 

dHa = —^aK A ■nKi 

by dS]), (gg]), we have 

= dlTa A W = ZaCLj /\UJ = ZaTTQ AlOq Auj. 

Thus zq = and 

(51) TTq = hu). 

When h =/= 0, we can define the ^-operator on S by defining *ujiq — lu2o, *uj2o — 
— cjio, and *1 = LUiQ A ^^20- 

The following definition is slightly more general than [5]: 

Definition 1. Let A/" — )■ A^"+'"(c) 6e an immersed submanifold of the space form. 
Let II be the second fundamental form of the submanifold. Let v be any normal 
vector of the submanifold. M" is called austere, if for any v and any < k < n/2, 
we have (J2k+i{i^ • II) — 0, where a'k{A) is the k-th elementary polynomial of the 
matrix A. 

In particular, if n = 3, austerity is equivalent to det(j/ • //) — and minimality. 

Corollary 2. If the equality in ConjectureUiis valid at any point and M is minimal, 
then M is an austere 3- fold. 

□ 

7. On CLASSIFICATION OF AuSTERE 3-SUBMANIFOLDS 

Austere 3-submanifolds of the Euclidean space and unit sphere were locally clas- 
sified in [Tj and [4], respectively. Their ideas can be used in the classification of 
austere 3-submanifolds of hyperbolic space. In this section, we will give the local 
classification of the submanifolds for which ([M]) hold in different space forms. 

Proposition 4. Using the above notations, we have 

VeoCo = 0. 

Thus locally an austere 3-fold is a fiber bundle over a 2-manifold and the fibers are 
geodesic lines of the space forms. More precisely, there is a neighborhood U of 
and smooth functions v,u : U — > R,3+r + |c| ^^^^ ^/j^,^ 
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(1) If c = 0, then M can be represented by v + tu; 

(2) If c ^ 0, then M can be represented by costv + sintu, \\u\\ — 1, = c, 
and V J- u with respect to the Riemannian or the Minkowski metric in (j42p , 
respectively. 

Proof. If Za = for all a, then M is totally geodesic and in this case the theorem 
follows easily. On the other hand, by (liT)) and we have VeoBo = on open set 
where at least one of Za 7^ 0. However, since M is minimal, all Za must be real an- 
alytic. Thus by continuity, the above is true everywhere. The local representations 
follows from the embeddings of S^'^'^ or H^'^'^ into R'*+'" or R^+'''^ respectively. 

□ 

Let E be the surface in M defined by {t = 0}. We define the complex structure 
J on S as follows: let P : TY, — > span{ei,e2} be the orthogonal projection. If 
Pe = ei, then we define Je = P^^e2. It is not hard to see that the definition only 
depends on the space {eo}^, not on the particular choices of ei, 62. 

Let ( ) J be the Riemannian metric of S such that {e, Je} forms an orthonormal 
frame. Let V, V"^ be the Levi-Civita connection with respect to the induced metric 
and ( )j, respectively. Let T = V-^ - V and let X = T(e, e) + T( Je, Je). Then 
we have the following: 

Theorem 6. The necessary and sufficient conditions for M to be an austere 3-fold 
are the following 

(1) //c = 0, then 

du £ spanju, dv}] 

if c ^ 0, then 

du G spanju, w, dv}; 

(2) // c = 0, then 

Av + Xv e N^M; 

if c ^ 0, then 

Av + Xv + Xv e Ny.M, 
where A = ||e|p -|- || Je|p, and N^M is the normal bundle o/S C M , which 
is a real line bundle. 

Proof. Let M be an austere 3-fold. Let II be the second fundamental form of 
M in the space forms. Since //(eo, ei) — II{eo, 62) = 0, we have (1); on the other 
hand, by minimality, let e = ei + fcieo, Je = e2 + ^269. Then we have 

(52) //(e, e) + //( Je, Je) = //(ei, ei) + //(ea, e^) - 0, 

which implies (2). 

Now we assume that (1), (2) are valid. Let {x,y) be a local coordinate system of 
E. If linear combinations of u, Vx, Vy {u, v, Vx,Vy, resp.), then for t small 

enough, Ux, Uy {—sint Vx+costux, — aintvy+costuy, resp.) are linear combinations 
of u,Vx + tux , Vy + tuy (— sin tv + cos t u, cos tvx + sin t Ux , cos tvy + sin t Uy , resp. ) . 
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Thus in a neighborhood of t, II{eQ,Y) ~ for any Y E TE. By the analyticity 
(which foUows from the minimahty) , //(eo, y) = is true whenever it is defined. 

Let Xq G M be a fixed point and let a{t) be the geodesic line passing through 
Xq such that <t'{0) — Cq. Let eo(i), ei(i), e2(i) be the parallel translation of the 
frames eg, ei, 62 along the geodesic cr(i). Then by the assumption that uJao = 0, we 
know eo{t), ei{t), e2{t) are parallel even in iV'^+''(c). Using this fact, we see that the 
equation ([52]), if true at one point xq, must be true on the geodesic a{t). Finally, 
(2) is equivalent to ((52|) . The theorem is proved. 

□ 

The above result gives a good classification of austere 3-folds. It is also possible to 
write out all the integrability conditions through the setting. However, as showed 
in [T], there are different integrablility conditions in different cases. For the sake 
of simplicity, for the rest of the paper, we will only locally classify all the generic 
austere 3-folds. The setting below is slightly different from that in Theorem[6l even 
though we still represent the 3-folds as v + tu, or costv + sintw, in what follows, 
the points {t = 0} may be singular points of the 3-folds. The classification was 
done in [I for c = and in 4 for c = 1 . 

The following result is from 1, Theorem 4.1], which solves the case c = 0: 

Theorem 7. If c = 0, and if the function h in ()5ip is not zero on M . Then locally 
M can be represented by v + tu for functions v,u: M ^ R'^+''. By replacing v with 
V — if necessary being a smooth function of Sj, we have 

(53) Am = -2u; 

(54) dv = ^dip u ^ ip * du 

where if satisfies — ~2ip. Here A is the Laplacian with respect to { ),/. 



The main result of this section is the following theorem. Note that when c = 1, it 
is known to Dajczer and Florit [H Theorem 14]. 

Theorem 8. Let c = ±1, and let the imaginary part of h be nonzero on M . Let 
M be the cone over M in the Euclidean space R*+'" (R?^^'^ , resp.). Then there 
is a densely open set M* of M such that M* H M is also densely open in M . 
Furthermore, M* can locally be represented by the regular points of the following 
map 

, 9g dg 
V + si— + S2-5-, 
ox oy 

where {x,y) is the local coordinate system of Yl and 11,(7 : E —> R^+'' (or R^^'''"'^, 
resp.) are smooth functions; Si,S2 are two real parameters. In addition, g satisfies 

Ag + X{g)^0, 

where X is a vector field and there are smooth functions 9,ip of T, such that 

dv — 9dg + If * dg, 



^In the case of c = 0, the 3-folds were locally completely classified. Refer to the paper for 
details. 
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and the functions 9, ip satisfy the following integrability conditions: 

(1) d0 = *{d<p-(pX*); 

(2) Atp - X{if ) - divXip = 0. 

Here X* is the 1 form on S which is dual to the vector field X with respect to the 
metric { ) j . 



The proof of the theorem wiU be similar to that in 4] in spirit. However, our proof 
is slightly simpler and we avoid using terms like cross sections or polar surfaces in 
that paper. Since the case of c = 1 is known and since the proof of c = 1,-1 are 
similar, we only give the proof of the theorem when c — —1. Recall that we embed 
the hyperbolic space form H^^"^ as a hypersurface in the Minkowski space R^^"^^^ 
with the metric in ([42]) . 

We first establish 

Proposition 5. // the imaginary part of h is nonzero, then there is a smooth 
function g : ^ R'^^'''-'^ such that 

(55) dg = tpiv + tp2eo, 

for some one forms "ipi, 7/12 on S. Note that Cq — u, where u is defined in Theorem\^ 



Proof. The integrability condition of (|55p is 

dipi V + d?/'2 Co - i/ii A (woeo + toici + ^^262) - ^2 A (wioei + ^2062 + ^af) = 0, 

which is equivalent to the following 

d^i - V2 A ti^o = 0; 
d^2 - V'l A ti^o = 0; 
■01 A + V'2 A uio = 0; 
Vi A ^2 + V'2 A a;2o = 0- 



(56) 



From the last two equations of the ([55)) , we get 

(57) - hi)2 + V-i = Ct^o 

for some complex function ^ on S. Then from ()57|) and the definition of the *- 
operator, we have 

(58) *(AV'2+^i)=BV^2, 

where h = — A— \J —\B and A, B arc smooth real functions of E. 



Since B ^ hy the assumption, the first two equations of ([56)) can be written as 

dip2 = {-Aip2 - B * V2) A Wo; 
* 02 = -g {-dB A *^2 - A ■02 + A{Ai/j2 + B * 02) A - V-'2 A wq)- 
Let ^0 be a solution of 

dip = {—Aijj2 — B * 1P2) A u!q. 
Then there is a smooth function /i such that 

^"2 = "0 + dn- 
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Insert the above equation into the equation of * -02 , we get 

(59) A/x + F(02) = O, 

where F{ip2) is a function of ■ip2 that contains the lower order terms. Thus (j59p and 
hence ([56|) are solvable locally. 

□ 

We define a complex structure on the space span{v, cq} by 

(60) J(eo - Av) = -Bv. 
We rewrite (f55|) as 

(61) = (i/'i+A02)t' + 02(eo-v4v). 
We have 

Lemma 4. On smooth l-forms ofT,, we have 

* = - J. 



Proof. From ((5T|) . have 

uio ^ -Aui + Buj2; 

UJ2Q = —BoJi — AuJ2. 

We let e € TS such that e = ei + fciCQ. Then by the definition of J on S, 
Je = 62 + k2eo. By ([51]) . we have 

J(wio)(e) = uJioiJe) = ^10(62) = B. 

On the other hand, 

*(a;io)(e) = W2o(e) = t^2o(ei) = -B, 
and the lemma is proved. 

□ 

Using the complex structure J on span {v,eo}, we have 
Lemma 5. Let Y be a smooth vector field ofE. Then 

dg{JY) = J{dg{Y)). 

Proof. Using (|6ip . ([SO]) , and Lemma 21 we have 

dg(jr) - (^i+A^2){JY)v+MJy)ieo-Av) = -BV;2(r)i;+^^-^(r)(eo-Az;). 
On the other side 

Jdg{Y) = J{{4;,+A^2){Y)v+My){eo~Av)) = tl±^{Y){eo-Av)-BMy)v- 
Thus dg(JY) = JdgiY) and the lemma is proved. 

□ 
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Proof of Theorem [H We let e e TE such that 

dg{e) = -a(eo - Av) 
for some smooth function a on S. Then by Lemma [S] we have 

dg{Je) = aBv. 

Let e = ei + /cieo. Then Je = 62 + feeo as m the proof of Lemma[4l We compute 
the span {61,62} component of 

Vedg[6) + Vjedg{J6). 

A straightforward computation shows that the it is equal to a(— VeiBo + A61 + 
B62) = by ([62]). Thus we conclude that 

(63) Ag + X{g) = 0. 

Since //(eo, Y) = II{v, Y) =Q for any Y G TM*, generically we have 

dv ~ dg o S 

for some endermorphism S : TS — > TE. From the integrable conditions, we know 
that S commutes with the second fundamental form //. Thus S* is a linear combi- 
nation of the identity morphism and the complex structure J, and there are smooth 
functions 0, (/3 of S such that 

(64) dv — ddg + tp * dg. 

Let dV — luiqAll!2o be the volume form of E. Then by a straightforward computation 
we have 

1* = - * {i{X)dV). 

Differentiating ((64)) ,we get 

de = 'i'idip- ipX*). 

Differentiating the above equation, we get 

Aip-X{(p)~divXif = 0. 

Finally, we observe that M* is defined to be the points of M such that {gx, gy,Vx + 
Sigxx + S2gxy,Vy + Si^xj/ + •S25j/j/ spaus a 4-dimensional space. By the genericity, 
M* is densely open in M. On the other hand, since x 1— > a;/||x|| is a submersion, 
M* n M is also densely open in M. The theorem is proved. 

□ 

In order to characterizing the submanifolds for which (|39p holds at every point, we 
make the following definition: 

Definition 2. W6 say a surface E R3+'~(R4+'', R3+'^'1 , resp.) satisfies property 
(A), if there is a complex structure J and a function w : E — s- R'^^"''(R^+'', R'^^'"'^, resp.) 
such that 
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du € span{u, dv}; 
du G span{u, w, dv}; 
Av + Xv e spanjw, dv}; 



(1) lfc=0, then 
if c =/= 0, then 

(2) // c = 0, then 
if c ^ Q, then 

Aw + Xv + Au € spanjit, dw}, 

where A = ||e|p + || Je|p; 

(3) There are orthonormal frames ^^i, • • • ,£,r in 'Rp'^'^ {S^'^'^ , H^'^'^ , resp.) which 
are perpendicular to span{u, dv} such that the second fundamental form of 
V on the , ^2 directions are 

and the second fundamental form on the directions (for i > 2) are zero, 
where A is an analytic function ofE. 



Combining Proposition [4] and Theorem [6l we have 

Theorem 9. Let Y, be a surface of property (A). Then there is an austere 3-fold 
M constructed in Propostion ^ satisfying (|39p . Conversely, any minimal 3-fold 
satisfying (|39p can be reconstructed using the functions v,u in Definition\^ through 
Proposition and Theorem O 



□ 
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